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Abstract 


Ethernet VPN (EVPN) provides an extensible and flexible multihoming VPN solution over an 
MPLS/IP network for intra-subnet connectivity among Tenant Systems and end devices that can 
be physical or virtual. However, there are scenarios for which there is a need for a dynamic and 
efficient inter-subnet connectivity among these Tenant Systems and end devices while 
maintaining the multihoming capabilities of EVPN. This document describes an Integrated 
Routing and Bridging (IRB) solution based on EVPN to address such requirements. 
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1. Introduction 


EVPN [RFC7432] provides an extensible and flexible multihoming VPN solution over an MPLS/IP 
network for intra-subnet connectivity among Tenant Systems (TSs) and end devices that can be 
physical or virtual, where an IP subnet is represented by an EVPN instance (EVI) for a VLAN- 
based service or by an (EVI, VLAN) association for a VLAN-aware bundle service. However, there 
are scenarios for which there is a need for a dynamic and efficient inter-subnet connectivity 
among these Tenant Systems and end devices while maintaining the multihoming capabilities of 
EVPN. This document describes an Integrated Routing and Bridging (IRB) solution based on EVPN 
to address such requirements. 


Inter-subnet communication is typically performed by centralized Layer 3 (L3) gateway (GW) 
devices, which enforce all inter-subnet communication policies and perform all inter-subnet 
forwarding. When two TSs belonging to two different subnets connected to the same Provider 
Edge (PE) wanted to communicate with each other, their traffic needed to be backhauled from the 
PE all the way to the centralized gateway where inter-subnet switching is performed and then 
sent back to the PE. For today's large multi-tenant Data Center (DC), this scheme is very inefficient 
and sometimes impractical. 
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In order to overcome the drawback of the centralized L3 GW approach, IRB functionality is 
needed on the PEs (also referred to as EVPN Network Virtualization Edges (NVEs)) attached to TSs 
in order to avoid inefficient forwarding of tenant traffic (i.e. avoid backhauling and hair 
pinning). When a PE with IRB capability receives tenant traffic over an Attachment Circuit (AC), it 
cannot only locally bridge the tenant intra-subnet traffic but also locally route the tenant inter- 
subnet traffic on a packet-by-packet basis, thus meeting the requirements for both intra- and 
inter-subnet forwarding and avoiding non-optimal traffic forwarding associated witha 
centralized L3 GW approach. 


Some TSs run non-IP protocols in conjunction with their IP traffic. Therefore, it is important to 
handle both kinds of traffic optimally -- e.g., to bridge non-IP and intra-subnet traffic and to route 
inter-subnet IP traffic. Therefore, the solution needs to meet the following requirements: 


R1: The solution must provide each tenant with IP routing of its inter-subnet traffic and Ethernet 
bridging of its intra-subnet traffic and non-routable traffic, where non-routable traffic refers 
to both non-IP traffic and IP traffic whose version differs from the IP version configured in IP 
Virtual Routing and Forwarding (IP-VRF). For example, if an IP-VRF in an NVEis configured for 
IPv6 and that NVE receives IPv4 traffic on the corresponding VLAN, then the IPv4 traffic is 
treated as non-routable traffic. 


R2: The solution must allow IP routing of inter-subnet traffic to be disabled on a per-VLAN basis 
on those PEs that are backhauling that traffic to another PE for routing. 


2. Terminology 


AC: Attachment Circuit 
ARP: Address Resolution Protocol 


ARP Table: A logical view of a forwarding table on a PE that maintains an IP to a MAC binding 
entry on an IP interface for both IPv4 and IPv6. These entries are learned through ARP/ 
ND or through EVPN. 


BD: Broadcast Domain. As per [RFC7432], an EVI consists of a single BD or multiple BDs. In 
the case of VLAN-bundle and VLAN-based service models (see [RFC7432]), a BD is 
equivalent to an EVI. In the case of a VLAN-aware bundle service model, an EVI 
contains multiple BDs. Also, in this document, "BD" and "subnet" are equivalent terms, 
and wherever "subnet" is used, it means "IP subnet". 


BD Route Target: Refers to the broadcast-domain-assigned Route Target [RFC4364]. In the case 
of a VLAN-aware bundle service model, all the BD instances in the MAC-VRF share the 
same Route Target. 


BT: Bridge Table. The instantiation of a BD in a MAC-VRF, as per [RFC7432]. 
CE: Customer Edge 
DA: Destination Address 
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Ethernet NVO Tunnel: Refers to Network Virtualization Overlay tunnels with an Ethernet 
payload, as specified for VXLAN in [RFC7348] and for NVGRE in [RFC7637]. 


EVI: EVPN Instance spanning NVE/PE devices that are participating on that EVPN, as per 
[RFC7432]. 


EVPN: Ethernet VPN, as per [RFC7432]. 


IP NVO Tunnel: Refers to Network Virtualization Overlay tunnels with IP payload (no MAC 
header in the payload) as specified for Generic Protocol Extension (GPE) in [VXLAN- 
GPE]. 


IP-VRF: A Virtual Routing and Forwarding table for IP routes on an NVE/PE. The IP routes could 
be populated by EVPN and IP-VPN address families. An IP-VRF is also an instantiation 
of a Layer 3 VPN in an NVE/PE. 


IRB: Integrated Routing and Bridging interface. It connects an IP-VRF to a BD (or subnet). 
MAC: Media Access Control 


MAC-VRF: A Virtual Routing and Forwarding table for MAC addresses on an NVE/PE, as per 
[RFC7432]. A MAC-VRF is also an instantiation of an EVI in an NVE/PE. 


ND: Neighbor Discovery 
NVE: Network Virtualization Edge 


NVGRE: Network Virtualization Using Generic Routing Encapsulation, as per [RFC7637]. 


NVO: Network Virtualization Overlay 

PE: Provider Edge 

RT-2: EVPN Route Type 2, i.e., MAC/IP Advertisement route, as defined in [RFC7432]. 

RT-5: EVPN Route Type 5, i.e., IP Prefix route, as defined in Section 3 of [RFC9136]. 

SA: Source Address 

TS: Tenant System 

VA: Virtual Appliance 

VNI: Virtual Network Identifier. As in [RFC8365], the term is used as a representation of a 24- 


bit NVO instance identifier, with the understanding that "VNI" will refer to a VXLAN 
Network Identifier in VXLAN, or a Virtual Subnet Identifier in NVGRE, etc., unless it is 
stated otherwise. 


VTEP: VXLAN Termination End Point, as per [RFC7348]. 


VXLAN: Virtual eXtensible Local Area Network, as per [RFC7348]. 


This document also assumes familiarity with the terminology of [RFC7365], [RFC7432], and 
[RFC8365]. 
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The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", 


"RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be 


interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all 


capitals, as shown here. 


3. EVPN PE Model for IRB Operation 


Since this document discusses IRB operation in relationship to EVPN MAC-VRF, IP-VRF, EVI, BD, 
bridge table, and IRB interfaces, it is important to understand the relationship between these 


components. Therefore, the PE model is illustrated below to a) describe these components and b) 
illustrate the relationship among them. 
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Figure 1: EVPN IRB PE Model 
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A tenant needing IRB services on a PE requires an IP-VRF table along with one or more MAC-VRF 


tables. An IP-VRE, as defined in [RFC4364], is the instantiation of an IP-VPN instance in a PE. A 
MAC-VRE as defined in [RFC7432], is the instantiation of an EVI in a PE. A MAC-VRF consists of 


one or more bridge tables, where each bridge table corresponds to a VLAN (broadcast domain). If 
service interfaces for an EVPN PE are configured in VLAN-based mode (i.e., Section 6.1 of 
[RFC7432]), then there is only a single bridge table per MAC-VRF (per EVI) -- i.e., there is only one 


tenant VLAN per EVI. However, if service interfaces for an EVPN PE are configured in VLAN- 


aware bundle mode (i.e., Section 6.3 of [RFC7432]), then there are several bridge tables per MAC- 


VRF (per EVI) -- i.e., there are several tenant VLANs per EVI. 
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Each bridge table is connected to an IP-VRF via an L3 interface called an "IRB interface". Since a 
single tenant subnet is typically (and in this document) represented by a VLAN (and thus 
supported by a single bridge table), for a given tenant, there are as many bridge tables as there 
are subnets. Thus, there are also as many IRB interfaces between the tenant IP-VRF and the 
associated bridge tables as shown in the PE model above. 


IP-VRF is identified by its corresponding Route Target and Route Distinguisher, and MAC-VRF is 
also identified by its corresponding Route Target and Route Distinguisher. If operating in EVPN 
VLAN-based mode, then a receiving PE that receives an EVPN route with a MAC-VRF Route Target 
can identify the corresponding bridge table; however, if operating in EVPN VLAN-aware bundle 
mode, then the receiving PE needs both the MAC-VRF Route Target and VLAN ID in order to 
identify the corresponding bridge table. 


4. Symmetric and Asymmetric IRB 


This document defines and describes two types of IRB solutions -- namely, symmetric and 
asymmetric IRB. The description of symmetric and asymmetric IRB procedures relating to data 
path operations and tables in this document is a logical view of data path lookups and related 
tables. Actual implementations, while following this logical view, may not strictly adhere to it for 
performance trade-offs. Specifically, 


e References to an ARP table in the context of asymmetric IRB is a logical view of a forwarding 
table that maintains an IP-to-MAC binding entry on a Layer 3 interface for both IPv4 and 
IPv6. These entries are not subject to ARP or ND protocols. For IP-to-MAC bindings learned via 
EVPN, an implementation may choose to import these bindings directly to the respective 
forwarding table (such as an adjacency/next-hop table) as opposed to importing them to ARP 
or ND protocol tables. 


e References to a host IP lookup followed by a host MAC lookup in the context of asymmetric 
IRB MAY be collapsed into a single IP lookup in a hardware implementation. 


In symmetric IRB, as its name implies, the lookup operation is symmetric at both the ingress and 
egress PEs -- i.e., both ingress and egress PEs perform lookups on both MAC and IP addresses. The 
ingress PE performs a MAC lookup followed by an IP lookup, and the egress PE performs an IP 
lookup followed by a MAC lookup, as depicted in the following figure. 
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Figure 2: Symmetric IRB 


In symmetric IRB, as shown in Figure 2, the inter-subnet forwarding between two PEs is done 
between their associated IP-VRFs. Therefore, the tunnel connecting these IP-VRFs can be either an 
IP-only tunnel (e.g., in the case of MPLS or GPE encapsulation) or an Ethernet NVO tunnel (e.g., in 
the case of VXLAN encapsulation). If it is an Ethernet NVO tunnel, the TS1's IP packet is 
encapsulated in an Ethernet header consisting of ingress and egress PE MAC addresses -- i.e., there 
is no need for the ingress PE to use the destination TS2's MAC address. Therefore, in symmetric 
IRB, there is no need for the ingress PE to maintain ARP entries for the association of the 
destination TS2's IP and MAC addresses in its ARP table. Each PE participating in symmetric IRB 
only maintains ARP entries for locally connected hosts and MAC-VRFs/BTs for only locally 
configured subnets. 


In asymmetric IRB, the lookup operation is asymmetric and the ingress PE performs three 
lookups, whereas the egress PE performs a single lookup --i.e., the ingress PE performs a MAC 
lookup, followed by an IP lookup, followed by a MAC lookup again. The egress PE performs just a 
single MAC lookup as depicted in Figure 3 below. 
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Figure 3: Asymmetric IRB 
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In asymmetric IRB, as shown in Figure 3, the inter-subnet forwarding between two PEs is done 
between their associated MAC-VRFs/BTs. Therefore, the MPLS or NVO tunnel used for inter-subnet 
forwarding MUST be of type Ethernet. Since only MAC lookup is performed at the egress PE (e.g., 
no IP lookup), the TS1's IP packets need to be encapsulated with the destination TS2's MAC 
address. In order for the ingress PE to perform such encapsulation, it needs to maintain TS2's IP 
and MAC address association in its ARP table. Furthermore, it needs to maintain destination TS2's 
MAC address in the corresponding bridge table even though it may not have any TSs of the 
corresponding subnet locally attached. In other words, each PE participating in asymmetric IRB 
MUST maintain ARP entries for remote hosts (hosts connected to other PEs) as well as maintain 
MAC-VRFs/BTs and IRB interfaces for ALL subnets in an IP-VRE including subnets that may not be 
locally attached. Therefore, careful consideration of the PE scale aspects for its ARP table size, its 
IRB interfaces, and the number and size of its bridge tables should be given for the application of 
asymmetric IRB. 


It should be noted that whenever a PE performs a host IP lookup for a packet that is routed, the 
IPv4 Time To Live (TTL) or IPv6 hop limit for that packet is decremented by one, and if it reaches 
zero, the packet is discarded. In the case of symmetric IRB, the TTL/ hop limit is decremented by 
both ingress and egress PEs (once by each), whereas in the case of asymmetric IRB, the TTL/ hop 
limit is decremented only once by the ingress PE. 


The following sections define the control and data plane procedures for symmetric and 
asymmetric IRB on ingress and egress PEs. The following figure is used to describe these 
procedures, showing a single IP-VRF and a number of BDs on each PE for a given tenant. That is, 
an IP-VRF connects one or more EVIs, and each EVI contains one MAC-VRF; each MAC VRF 
consists of one or more bridge tables, one per BD; anda PE has an associated IRB interface for 
each BD. 


PE 1 pees + 
poasecsaecacee + | | 
TS1----- | MACx | | | PE2 
(M1/IP1) |(BT1) | | [| oo tea----------- + 
TS5----- | \ | | MPLS/ | |MACy (BT3) |----- TS3 
(M5/IP5) |IPx/Mx \ | | VXLAN/ | / | (M3/IP3) 
| (IP-VRF1)|----| NVGRE |---|(IP-VRF1) | 
| | | | \ | 
TS2----- |(BT2) / | | Ja T (BTID |----- TS4 
(M2/IP2) | | | (arn | (M4/IP4) 
+------------- + l l +------------- + 
| | 
+--------- + 


Figure 4: IRB Forwarding 
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4.1. IRB Interface and Its MAC and IP Addresses 


To support inter-subnet forwarding on a PE, the PE acts as an IP default gateway from the 
perspective of the attached Tenant Systems where default gateway MAC and IP addresses are 
configured on each IRB interface associated with its subnet and fall into one of the following two 
options: 


1. All the PEs for a given tenant subnet use the same anycast default gateway IP and MAC 
addresses. On each PE, these default gateway IP and MAC addresses correspond to the IRB 
interface connecting the bridge table associated with the tenant's VLAN to the corresponding 
tenant's IP-VRF. 


2. Each PE for a given tenant subnet uses the same anycast default gateway IP address but its 
own MAC address. These MAC addresses are aliased to the same anycast default gateway IP 
address through the use of the Default Gateway extended community as specified in 
[RFC7432], which is carried in the EVPN MAC/IP Advertisement routes. On each PE, this default 
gateway IP address, along with its associated MAC addresses, correspond to the IRB interface 
connecting the bridge table associated with the tenant's VLAN to the corresponding tenant's 
IP-VRF. 


It is worth noting that if the applications that are running on the TSs are employing or relying on 
any form of MAC security, then the first option (i.e., using an anycast MAC address) should be used 
to ensure that the applications receive traffic from the same IRB interface MAC address to which 
they are sending. If the second option is used, then the IRB interface MAC address MUST be the 
one used in the initial ARP reply or ND Neighbor Advertisement (NA) for that TS. 


Although both of these options are applicable to both symmetric and asymmetric IRB, option 1 is 
recommended because of the ease of anycast MAC address provisioning on not only the IRB 
interface associated with a given subnet across all the PEs corresponding to that VLAN but also 
on all IRB interfaces associated with all the tenant's subnets across all the PEs corresponding to 
all the VLANs for that tenant. Furthermore, it simplifies the operation as there is no need for 
Default Gateway extended community advertisement and its associated MAC aliasing procedure. 
Yet another advantage is that following host mobility, the host does not need to refresh the default 
GW ARP/ND entry. 


If option 1 is used, an implementation MAY choose to auto-derive the anycast MAC address. If 
auto-derivation is used, the anycast MAC MUST be auto-derived out of the following ranges (which 
are defined in [RFC5798]): 


e Anycast IPv4 IRB case: 00-00-5E-00-01-{VRID} 
e Anycast IPv6 IRB case: 00-00-5E-00-02-{VRID} 


Where the last octet is generated based on a configurable Virtual Router ID (VRID) (range 1-255). 
If not explicitly configured, the default value for the VRID octet is '1'. Auto-derivation of the 
anycast MAC can only be used if there is certainty that the auto-derived MAC does not collide with 
any customer MAC address. 
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In addition to IP anycast addresses, IRB interfaces can be configured with non-anycast IP 
addresses for the purpose of OAM (such as sending a traceroute/ping to these interfaces) for both 
symmetric and asymmetric IRB. These IP addresses need to be distributed as VPN routes when 
PEs operate in symmetric IRB mode. However, they don't need to be distributed if the PEs are 
operating in asymmetric IRB mode as the non-anycast IP addresses are configured along with 
their individual MACs, and they get distributed via the EVPN route type 2 advertisement. 


For option 1 — irrespective of whether only the anycast MAC address or both anycast and non- 
anycast MAC addresses (where the latter one is used for the purpose of OAM) are used on the 
same IRB -- when a TS sends an ARP request or ND Neighbor Solicitation (NS) to the PE to which it 
is attached, the request is sent for the anycast IP address of the IRB interface associated with the 
TS's subnet. The reply will use an anycast MAC address (in both the source MAC in the Ethernet 
header and sender hardware address in the payload). For example, in Figure 4, TS1 is configured 
with the anycast IPx address as its default gateway IP address; thus, when it sends an ARP request 
for IPx (anycast IP address of the IRB interface for BT1), the PE1 sends an ARP reply with the 
MACXx, which is the anycast MAC address of that IRB interface. Traffic routed from IP-VRF1 to TS1 
uses the anycast MAC address as the source MAC address. 


4.2. Operational Considerations 


Symmetric and asymmetric IRB modes may coexist in the same network, and an ingress PE that 
supports both forwarding modes for a given tenant can interwork with egress PEs that support 
either IRB mode. The egress PE will indicate the desired forwarding mode for a given host based 
on the presence of the Label2 field and the IP-VRF Route Target in the EVPN MAC/IP 
Advertisement route. If the Label2 field of the received MAC/IP Advertisement route for host H1 is 
non-zero, and one of its Route Targets identifies the IP-VRF, the ingress PE will use symmetric IRB 
mode when forwarding packets destined to H1. If the Label2 field is zero and the MAC/IP 
Advertisement route for H1 does not carry any Route Target that identifies the IP-VRF the ingress 
PE will use asymmetric mode when forwarding traffic to H1. 


As an example that illustrates the previous statement, suppose PE1 and PE2 need to forward 
packets from TS2 to TS4 in Figure 4. Since both PEs are attached to the bridge table of the 
destination host, symmetric and asymmetric IRB modes are both possible as long as the ingress 
PE, PE1, supports both modes. The forwarding mode will depend on the mode configured in the 
egress PE, PE2. That is: 


1. If PE2 is configured for symmetric IRB mode, PE2 will advertise TS4 MAC/IP addresses in a 
MAC/IP Advertisement route with a non-zero Label2 field, e.g., Label2 = Lx, and a Route Target 
that identifies IP-VRF1 in PE1. IP4 will be installed in PE1's IP-VRF1; TS4's ARP and MAC 
information will also be installed in PE1's IRB interface ARP table and BT1, respectively. When 
a packet from TS2 destined to TS4 is looked up in PE1's IP-VRF route table, a longest prefix 
match lookup will find IP4 in the IP-VRF, and PE1 will forward using the symmetric IRB mode 
and Label Lx. 


2. However, if PE2 is configured for asymmetric IRB mode, PE2 will advertise TS4 MAC/IP 
information in a MAC/IP Advertisement route with a zero Label2 field and no Route Target 
identifying IP-VRF1. In this case, PE2 will install TS4 information in its ARP table and BT1. 
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When a packet from TS2 to TS4 arrives at PE1, a longest prefix match on IP-VRF1's route table 
will yield the local IRB interface to BT1, where a subsequent ARP and bridge table lookup will 
provide the information for an asymmetric forwarding mode to PE2. 


Refer to [EVPN] for more information about interoperability between symmetric and 
asymmetric forwarding modes. 


The choice between symmetric or asymmetric mode is based on the operator's preference, and it 
is a trade-off between scale (which is better in the symmetric IRB mode) and control plane 
simplicity (asymmetric IRB mode simplifies the control plane). In cases where a tenant has hosts 
for every subnet attached to all (or most of) the PEs, the ARP and MAC entries need to be learned 
by all PEs anyway; therefore, the asymmetric IRB mode simplifies the forwarding model and 
saves space in the IP-VRF route table, since host routes are not installed in the route table. 
However, if the tenant does not need to stretch subnets (broadcast domains) to multiple PEs and 
inter-subnet forwarding is needed, the symmetric IRB model will save ARP and bridge table space 
in all the PEs (in comparison with the asymmetric IRB model). 


5. Symmetric IRB Procedures 


5.1. Control Plane - Advertising PE 


When a PE (e.g., PE1 in Figure 4 above) learns the MAC and IP address of a TS (e.g., via an ARP 
request or Neighbor Solicitation), it adds the MAC address to the corresponding MAC-VRF/BT of 
that tenant's subnet and adds the IP address to the IP-VRF for that tenant. Furthermore, it adds 
this TS's MAC and IP address association to its ARP table or Neighbor Discovery Protocol (NDP) 
cache. It then builds an EVPN MAC/IP Advertisement route (type 2) as follows and advertises it to 
other PEs participating in that tenant's VPN. 


e The Length field of the BGP EVPN Network Layer Reachability Information (NLRI) for an 
EVPN MAC/IP Advertisement route MUST be either 40 (if the IPv4 address is carried) or 52 (if 
the IPv6 address is carried). 

e The Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet Tag ID, MAC Address 
Length, MAC Address, IP Address Length, IP Address, and MPLS Label! fields MUST be set per 
[RFC7432] and [RFC8365]. 

e The MPLS Label2 field is set to either an MPLS label or a VNI corresponding to the tenant's IP- 
VRE In the case of an MPLS label, this field is encoded as 3 octets, where the high-order 20 bits 
contain the label value. 


Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length, MAC Address, IP Address 
Length, and IP Address fields are part of the route key used by BGP to compare routes. The rest of 
the fields are not part of the route key. 


This route is advertised along with the following two extended communities: 


1. Encapsulation Extended Community 
2. EVPN Router's MAC Extended Community 
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This route is advertised with one or more Encapsulation Extended Communities [RFC9012], one 
for each encapsulation type supported by the advertising PE. If one or more encapsulation types 
require an Ethernet frame, a single EVPN Router's MAC Extended Community (Section 8.1) is also 
advertised. This extended community specifies the MAC address to be used as the inner 
destination MAC address in an Ethernet frame sent to the advertising PE. 


This route MUST be advertised with two Route Targets, one corresponding to the MAC-VRF of the 
tenant's subnet and another corresponding to the tenant's IP-VRF. 


5.2. Control Plane - Receiving PE 


When a PE (e.g., PE2 in Figure 4 above) receives this EVPN MAC/IP Advertisement route, it 
performs the following: 


e The MAC-VRF Route Target and Ethernet Tag, if the latter is non-zero, are used to identify the 
correct MAC-VRF and bridge table, and if they are found, the MAC address is imported. The IP- 
VRF Route Target is used to identify the correct IP-VRF, and if it is found, the IP address is 
imported. 


If the MPLS Label2 field is non-zero, it means that this route is to be used for symmetric IRB, and 
the MPLS label2 value is to be used when sending a packet for this IP address to the advertising PE. 


If the receiving PE supports asymmetric IRB mode and receives this route with both the MAC-VRF 
and IP-VRF Route Targets but the MAC/IP Advertisement route does not include the MPLS Label2 
field, then the receiving PE installs the MAC address in the corresponding MAC-VRF and the (IP, 
MAC) association in the ARP table for that tenant (identified by the corresponding IP-VRF Route 
Target). 


If the receiving PE receives this route with both the MAC-VRF and IP-VRF Route Targets, and if the 
receiving PE does not support either asymmetric or symmetric IRB modes but has the 
corresponding MAC-VRE then it only imports the MAC address. 


If the receiving PE receives this route with both the MAC-VRF and IP-VRF Route Targets and the 
MAC/IP Advertisement route includes the MPLS Label2 field but the receiving PE only supports 

asymmetric IRB mode, then the receiving PE MUST ignore the MPLS Label? field and install the 
MAC address in the corresponding MAC-VRF and (IP, MAC) association in the ARP table for that 
tenant (identified by the corresponding IP-VRF Route Target). 


5.3. Subnet Route Advertisement 


In the case of symmetric IRB, a Layer 3 subnet and IRB interface corresponding to a MAC-VRF/BT 
are required to be provisioned at a PE only if that PE has locally attached hosts in that subnet. In 
order to enable inter-subnet routing across PEs in a deployment where not all subnets are 
provisioned at all PEs participating in an EVPN IRB instance, PEs MUST advertise local subnet 
routes as EVPN RT-5. These subnet routes are required for bootstrapping host (IP, MAC) learning 
using gleaning procedures initiated by an inter-subnet data packet. 
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That is, if a given host's (IP, MAC) association is unknown, and an ingress PE needs to senda 
packet to that host, then that ingress PE needs to know which egress PEs are attached to the 
subnet in which the host resides in order to send the packet to one of those PEs, causing the PE 
receiving the packet to probe for that host. For example, consider a subnet A that is locally 
attached to PE1 and subnet B that is locally attached to PE2 and PE3. Host A in subnet A, which is 
attached to PE1, initiates a data packet destined to host B in subnet B, which is attached to PE3. If 
host B's (IP, MAC) has not yet been learned via either a gratuitous ARP OR a prior gleaning 
procedure, a new gleaning procedure MUST be triggered for host B's (IP MAC) to be learned and 
advertised across the EVPN network. Since host B's subnet is not local to PE1, an IP lookup for 
host B at PE1 will not trigger this gleaning procedure for host B's (IP, MAC). Therefore, PE1 MUST 
learn subnet B's prefix route via EVPN RT-5 advertised from PE2 and PE3, so it can route the 
packet to one of the PEs that have subnet B locally attached. Once the packet is received at PE2 
OR PE3, and the route lookup yields a glean result, an ARP request is triggered and flooded across 
the Layer 2 overlay. This ARP request would be received and replied to by host B, resulting in host 
B (IP, MAC) learning at PE3 and its advertisement across the EVPN network. Packets from host A 
to host B can now be routed directly from PE1 to PE3. Advertisement of local subnet EVPN RT-5 for 
an IP-VRF MAY typically be achieved via provisioning-connected route redistribution to BGP. 


5.4. Data Plane - Ingress PE 


When an Ethernet frame is received by an ingress PE (e.g., PE1 in Figure 4 above), the PE uses the 
AC ID (e.g., VLAN ID) to identify the associated MAC-VRF/BT, and it performs a lookup on the 
destination MAC address. If the MAC address corresponds to its IRB interface MAC address, the 
ingress PE deduces that the packet must be inter-subnet routed. Hence, the ingress PE performs an 
IP lookup in the associated IP-VRF table. The lookup identifies the BGP next hop of the egress PE 
along with the tunnel/encapsulation type and the associated MPLS/VNI values. The ingress PE 
also decrements the TTL / hop limit for that packet by one, and if it reaches zero, the ingress PE 
discards the packet. 


If the tunnel type is that of an MPLS or IP-only NVO tunnel, then the TS's IP packet is sent over the 
tunnel without any Ethernet header. However, if the tunnel type is that of an Ethernet NVO 
tunnel, then an Ethernet header needs to be added to the TS's IP packet. The source MAC address 
of this inner Ethernet header is set to the ingress PE's router MAC address, and the destination 
MAC address of this inner Ethernet header is set to the egress PE's router MAC address learned via 
the EVPN Router's MAC Extended Community attached to the route. The MPLS VPN label is set to 
the received label2 in the route. In the case of the Ethernet NVO tunnel type, the VNI may be set 
one of two ways: 


downstream mode: The VNIis set to the received label2 in the route, which is downstream 
assigned. 


globalmode: The VNTis set to the received label2 in the route, which is assigned domain-wide. 
This VNI value from the received label2 MUST be the same as the locally configured VNI for the 
IP-VRF as all PEs in the NVO MUST be configured with the same IP-VRF VNI for this mode of 
operation. If the received label2 value does not match the locally configured VNI value, the 
route MUST NOT be used, and an error message SHOULD be logged. 
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PEs may be configured to operate in one of these two modes depending on the administrative 
domain boundaries across PEs participating in the NVO and the PE's capability to support 
downstream VNI mode. 


In the case of NVO tunnel encapsulation, the outer source and destination IP addresses are set to 
the ingress and egress PE BGP next-hop IP addresses, respectively. 


5.5. Data Plane - Egress PE 


When the tenant's MPLS or NVO encapsulated packet is received over an MPLS or NVO tunnel by 
the egress PE, the egress PE removes the NVO tunnel encapsulation and uses the VPN MPLS label 
(for MPLS encapsulation) or VNI (for NVO encapsulation) to identify the IP-VRF in which IP 
lookup needs to be performed. If the VPN MPLS label or VNI identifies a MAC-VRF instead of an IP- 
VRE, then the procedures in Section 6.4 for asymmetric IRB are executed. 


The lookup in the IP-VRF identifies a local adjacency to the IRB interface associated with the 
egress subnet's MAC-VRF/BT. The egress PE also decrements the TTL/ hop limit for that packet by 
one, and if it reaches zero, the egress PE discards the packet. 


The egress PE gets the destination TS's MAC address for that TS's IP address from its ARP table or 
NDP cache. It encapsulates the packet with that destination MAC address and a source MAC 
address corresponding to that IRB interface and sends the packet to its destination subnet MAC- 
VRF/BT. 


The destination MAC address lookup in the MAC-VRF/BT results in the local adjacency (e.g., local 
interface) over which the Ethernet frame is sent. 


6. Asymmetric IRB Procedures 


6.1. Control Plane - Advertising PE 


When a PE (e.g., PE1 in Figure 4 above) learns the MAC and IP address of an attached TS (e.g., via 
an ARP request or ND Neighbor Solicitation), it populates its MAC-VRF/BT, IP-VRF, and ARP table or 
NDP cache just as in the case for symmetric IRB. It then builds an EVPN MAC/IP Advertisement 
route (type 2) as follows and advertises it to other PEs participating in that tenant's VPN. 


e The Length field of the BGP EVPN NLRI for an EVPN MAC/IP Advertisement route MUST be 
either 37 Gif an IPv4 address is carried) or 49 (if an IPv6 address is carried). 


e The RD, Ethernet Segment Identifier, Ethernet Tag ID, MAC Address Length, MAC Address, IP 
Address Length, IP Address, and MPLS Label! fields MUST be set per [RFC7432] and [RFC8365]. 


e The MPLS Label2 field MUST NOT be included in this route. 


Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length, MAC Address, IP Address 
Length, and IP Address fields are part of the route key used by BGP to compare routes. The rest of 
the fields are not part of the route key. 
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This route is advertised along with the following extended community: 
e Tunnel Type Extended Community 


For asymmetric IRB mode, the EVPN Router's MAC Extended Community is not needed because 
forwarding is performed using destination TS's MAC address, which is carried in this EVPN route 
type 2 advertisement. 


This route MUST always be advertised with the MAC-VRF Route Target. It MAY also be advertised 
with a second Route Target corresponding to the IP-VRF. 


6.2. Control Plane - Receiving PE 


When a PE (e.g., PE2 in Figure 4 above) receives this EVPN MAC/IP Advertisement route, it 
performs the following: 


e Using the MAC-VRF Route Target, it identifies the corresponding MAC-VRF and imports the 
MAC address into it. For asymmetric IRB mode, it is assumed that all PEs participating in a 
tenant's VPN are configured with all subnets (i.e., all VLANs) and corresponding MAC-VRFs/ 
BTs even if there are no locally attached TSs for some of these subnets. This is because the 
ingress PE needs to do forwarding based on the destination TS's MAC address and perform 
NVO tunnel encapsulation as the property of a lookup in the MAC-VRF/BT. 


e If only the MAC-VRF Route Target is used, then the receiving PE uses the MAC-VRF Route 
Target to identify the corresponding IP-VRF -- i.e., many MAC-VRF Route Targets map to the 
same IP-VRF for a given tenant. In this case, MAC-VRF may be used by the receiving PE to 
identify the corresponding IP-VRF via the IRB interface associated with the subnet MAC-VRF/ 
BT. In this case, the MAC-VRF Route Target may be used by the receiving PE to identify the 
corresponding IP-VRF. 


e Using the MAC-VRF Route Target, the receiving PE identifies the corresponding ARP table or 
NDP cache for the tenant, and it adds an entry to the ARP table or NDP cache for the TS's MAC 
and IP address association. It should be noted that the tenant's ARP table or NDP cache at the 
receiving PE is identified by all the MAC-VRF Route Targets for that tenant. 

e If the IP-VRF Route Target is included, it may be used to import the route to IP-VRF. If the IP- 
VRF Route Target is not included, MAC-VRF is used to derive the corresponding IP-VRF for 
import, as explained in the prior section. In both cases, an IP-VRF route is installed with the TS 
MAC binding included in the received route. 


If the receiving PE receives the MAC/IP Advertisement route with the MPLS Label2 field but the 
receiving PE only supports asymmetric IRB mode, then the receiving PE MUST ignore the MPLS 
Label2 field and install the MAC address in the corresponding MAC-VRF and (IP, MAC) association 
in the ARP table or NDP cache for that tenant (with the IRB interface identified by the MAC-VRF). 


6.3. Data Plane - Ingress PE 


When an Ethernet frame is received by an ingress PE (e.g., PE1 in Figure 4 above), the PE uses the 
AC ID (e.g., VLAN ID) to identify the associated MAC-VRF/BT, and it performs a lookup on the 
destination MAC address. If the MAC address corresponds to its IRB interface MAC address, the 
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ingress PE deduces that the packet must be inter-subnet routed. Hence, the ingress PE performs an 
IP lookup in the associated IP-VRF table. The lookup identifies a local adjacency to the IRB 
interface associated with the egress subnet's MAC-VRF/ bridge table. The ingress PE also 
decrements the TTL / hop limit for that packet by one, and if it reaches zero, the ingress PE 
discards the packet. 


The ingress PE gets the destination TS's MAC address for that TS's IP address from its ARP table or 
NDP cache. It encapsulates the packet with that destination MAC address and a source MAC 
address corresponding to that IRB interface and sends the packet to its destination subnet MAC- 
VRF/BT. 


The destination MAC address lookup in the MAC-VRF/BT results in a BGP next-hop address of the 
egress PE along with label1 (L2 VPN MPLS label or VNI). The ingress PE encapsulates the packet 
using the Ethernet NVO tunnel of the choice (e.g., VXLAN or NVGRE) and sends the packet to the 
egress PE. Because the packet forwarding is between the ingress PE's MAC-VRF/BT and the egress 
PE's MAC-VRF/ bridge table, the packet encapsulation procedures follow that of [RFC7432] for 
MPLS and [RFC8365] for VXLAN encapsulations. 


6.4. Data Plane - Egress PE 


When a tenant's Ethernet frame is received over an NVO tunnel by the egress PE, the egress PE 
removes the NVO tunnel encapsulation and uses the VPN MPLS label (for MPLS encapsulation) or 
VNI (for NVO encapsulation) to identify the MAC-VRF/BT in which the MAC lookup needs to be 
performed. 


The MAC lookup results in a local adjacency (e.g., local interface) over which the packet needs to 
get sent. 


Note that the forwarding behavior on the egress PE is the same as the EVPN intra-subnet 
forwarding described in [RFC7432] for MPLS and [RFC8365] for NVO networks. In other words, all 
the packet processing associated with the inter-subnet forwarding semantics is confined to the 
ingress PE for asymmetric IRB mode. 


It should also be noted that [RFC7432] provides a different level of granularity for the EVPN label. 
Besides identifying the bridge domain table, it can be used to identify the egress interface ora 
destination MAC address on that interface. If an EVPN label is used for an egress interface or 
individual MAC address identification, then no MAC lookup is needed in the egress PE for MPLS 
encapsulation, and the packet can be directly forwarded to the egress interface just based on the 
EVPN label lookup. 


7. Mobility Procedure 


When a TS moves from one NVE (aka source NVE) to another NVE (aka target NVE), it is 
important that the MAC Mobility procedures be properly executed and the corresponding MAC- 
VRF and IP-VRF tables on all participating NVEs be updated. [RFC7432] describes the MAC 
Mobility procedures for L2-only services for both single-homed TS and multihomed TS. This 
section describes the incremental procedures and BGP Extended Communities needed to handle 
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the MAC Mobility for IRB. In order to place the emphasis on the differences between L2-only and 
IRB use cases, the incremental procedure is described for a single-homed TS with the expectation 
that the additional steps needed for a multihomed TS can be extended per Section 15 of [RFC7432]. 
This section describes mobility procedures for both symmetric and asymmetric IRB. Although the 
language used in this section is for IPv4 ARP, it equally applies to IPv6 ND. 


When a TS moves from a source NVE to a target NVE, it can behave in one of the following three 
ways: 


1. TS initiates an ARP request upon a move to the target NVE. 
2.TS sends a data packet without first initiating an ARP request to the target NVE. 
3. TS is a silent host and neither initiates an ARP request nor sends any packets. 


Depending on the expected TS's behavior, an NVE needs to handle at least the first option and 
should be able to handle the second and third options. The following subsections describe the 
procedures for each scenario where it is assumed that the MAC and IP addresses of a TS havea 
one-to-one relationship (i.e., there is one IP address per MAC address and vice versa). The 
procedures for host mobility detection in the presence of a many-to-one relationship is outside 
the scope of this document, and it is covered in [EXTENDED-MOBILITY]. The "many-to-one 
relationship" refers to many host IP addresses corresponding to a single host MAC address or 
many host MAC addresses corresponding to a single IP address. It should be noted that in the case 
of IPv6, a link-local IP address does not count in a many-to-one relationship because that address 
is confined to a single Ethernet segment, and it is not used for host mobility (i.e., by definition, 
host mobility is between two different Ethernet segments). Therefore, when an IPv6 host is 
configured with both a Global Unicast address (or a Unique Local address) and a link-local 
address, for the purpose of host mobility, it is considered with a single IP address. 


7.1. Initiating a Gratuitous ARP upon a Move 


In this scenario, when a TS moves from a source NVE to a target NVE, the TS initiates a gratuitous 
ARP upon the move to the target NVE. 


The target NVE, upon receiving this ARP message, updates its MAC-VRE, IP-VRF, and ARP table with 
the host MAC, IP, and local adjacency information (e.g., local interface). 


Since this NVE has previously learned the same MAC and IP addresses from the source NVE, it 
recognizes that there has been a MAC move, and it initiates MAC Mobility procedures per 
[RFC7432] by advertising an EVPN MAC/IP Advertisement route with both the MAC and IP 
addresses filled in (per Sections 5.1 and 6.1) along with the MAC Mobility extended community, 
with the sequence number incremented by one. The target NVE also exercises the MAC 
duplication detection procedure in Section 15.1 of [RFC7432]. 


The source NVE, upon receiving this MAC/IP Advertisement route, realizes that the MAC has 
moved to the target NVE. It updates its MAC-VRF and IP-VRF table accordingly with the adjacency 
information of the target NVE. In the case of the asymmetric IRB, the source NVE also updates its 
ARP table with the received adjacency information, and in the case of the symmetric IRB, the 
source NVEremoves the entry associated with the received (IP, MAC) from its local ARP table. It 
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then withdraws its EVPN MAC/IP Advertisement route. Furthermore, it sends an ARP probe locally 
to ensure that the MAC is gone. If an ARP response is received, the source NVE updates its ARP 
entry for that (IP, MAC) and re-advertises an EVPN MAC/IP Advertisement route for that (IP MAC) 
along with the MAC Mobility extended community, with the sequence number incremented by 
one. The source NVE also exercises the MAC duplication detection procedure in Section 15.1 of 
[RFC7432]. 


All other remote NVE devices, upon receiving the MAC/IP Advertisement route with the MAC 
Mobility extended community, compare the sequence number in this advertisement with the one 
previously received. If the new sequence number is greater than the old one, then they update the 
MAC/IP addresses of the TS in their corresponding MAC-VRF and IP-VRF tables to point to the 
target NVE. Furthermore, upon receiving the MAC/IP withdraw for the TS from the source NVE, 
these remote PEs perform the cleanups for their BGP tables. 


7.2. Sending Data Traffic without an ARP Request 


In this scenario, when a TS moves from a source NVE to a target NVE, the TS starts sending data 
traffic without first initiating an ARP request. 


The target NVE, upon receiving the first data packet, learns the MAC address of the TS in the data 
plane and updates its MAC-VRF table with the MAC address and the local adjacency information 
(e.g., local interface) accordingly. The target NVE realizes that there has been a MAC move 
because the same MAC address has been learned remotely from the source NVE. 


If EVPN-IRB NVEs are configured to advertise MAC-only routes in addition to MAC-and-IP EVPN 
routes, then the following steps are taken: 


e The target NVE, upon learning this MAC address in the data plane, updates this MAC address 
entry in the corresponding MAC-VRF with the local adjacency information (e.g., local 
interface). It also recognizes that this MAC has moved and initiates MAC Mobility procedures 
per [RFC7432] by advertising an EVPN MAC/IP Advertisement route with only the MAC 
address filled in along with the MAC Mobility extended community, with the sequence 
number incremented by one. 


e The source NVE, upon receiving this MAC/IP Advertisement route, realizes that the MAC has 
moved to the new NVE. It updates its MAC-VRF table with the adjacency information for that 
MAC address to point to the target NVE and withdraws its EVPN MAC/IP Advertisement route 
that has only the MAC address (if it has advertised such a route previously). Furthermore, it 
searches for the corresponding MAC-IP entry and sends an ARP probe for this (IP, MAC) pair. 
The ARP request message is sent both locally to all attached TSs in that subnet as well as to 
other NVEs participating in that subnet, including the target NVE. Note that the PE needs to 
maintain a correlation between MAC and MAC-IP route entries in the MAC-VRF to 
accomplish this. 


e The target NVE passes the ARP request to its locally attached TSs, and when it receives the ARP 
response, it updates its IP-VRF and ARP table with the host (IPR MAC) information. It also sends 
an EVPN MAC/IP Advertisement route with both the MAC and IP addresses filled in along with 
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the MAC Mobility extended community, with the sequence number set to the same value as 
the one for the MAC-only Advertisement route it sent previously. 


e When the source NVE receives the EVPN MAC/IP Advertisement route, it updates its IP-VRF 
table with the new adjacency information (pointing to the target NVE). In the case of the 
asymmetric IRB, the source NVE also updates its ARP table with the received adjacency 
information, and in the case of the symmetric IRB, the source NVE removes the entry 
associated with the received (IP, MAC) from its local ARP table. Furthermore, it withdraws its 
previously advertised EVPN MAC/IP route with both the MAC and IP address fields filled in. 


e All other remote NVE devices, upon receiving the MAC/IP Advertisement route with the MAC 
Mobility extended community, compare the sequence number in this advertisement with the 
one previously received. If the new sequence number is greater than the old one, then they 
update the MAC/IP addresses of the TS in their corresponding MAC-VRF, IP-VRF, and ARP 
tables (in the case of asymmetric IRB) to point to the new NVE. Furthermore, upon receiving 
the MAC/IP withdraw for the TS from the old NVE, these remote PEs perform the cleanups for 
their BGP tables. 


If an EVPN-IRB NVE is configured not to advertise MAC-only routes, then upon receiving the first 
data packet, it learns the MAC address of the TS and updates the MAC entry in the corresponding 
MAC-VRF table with the local adjacency information (e.g., local interface). It also realizes that 
there has been a MAC move because the same MAC address has been learned remotely from the 
source NVE. It uses the local MAC route to find the corresponding local MAC-IP route and sends a 
unicast ARP request to the host. When receiving an ARP response, it follows the procedure 
outlined in Section 7.1. In the prior case, where MAC-only routes are also advertised, this 
procedure of triggering a unicast ARP probe at the target PE MAY also be used in addition to the 
source PE broadcast ARP probing procedure described earlier for better convergence. 


7.3. Silent Host 


In this scenario, when a TS moves from a source NVE to a target NVE, the TS is silent, and it 
neither initiates an ARP request nor sends any data traffic. Therefore, neither the target nor the 
source NVEs are aware of the MAC move. 


On the source NVE, an age-out timer (for the silent host that has moved) is used to trigger an ARP 
probe. This age-out timer can be either an ARP timer or a MAC age-out timer, and this is an 
implementation choice. The ARP request gets sent both locally to all the attached TSs on that 
subnet as well as to all the remote NVEs (including the target NVE) participating in that subnet. 
The source NVE also withdraws the EVPN MAC/IP Advertisement route with only the MAC address 
(if it has previously advertised such a route). 


The target NVE passes the ARP request to its locally attached TSs, and when it receives the ARP 
response, it updates its MAC-VRF, IP-VRE, and ARP table with the host (IP, MAC) and local 
adjacency information (e.g., local interface). It also sends an EVPN MAC/IP Advertisement route 
with both the MAC and IP address fields filled in along with the MAC Mobility extended 
community, with the sequence number incremented by one. 
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When the source NVE receives the EVPN MAC/IP Advertisement route, it updates its IP-VRF table 
with the new adjacency information (pointing to the target NVE). In the case of the asymmetric 
IRB, the source NVE also updates its ARP table with the received adjacency information, and in 
the case of the symmetric IRB, the source NVE removes the entry associated with the received (IP, 
MAC) from its local ARP table. Furthermore, it withdraws its previously advertised EVPN MAC/IP 
route with both the MAC and IP address fields filled in. 


All other remote NVE devices, upon receiving the MAC/IP Advertisement route with the MAC 
Mobility extended community, compare the sequence number in this advertisement with the one 
previously received. If the new sequence number is greater than the old one, then they update the 
MAC/IP addresses of the TS in their corresponding MAC-VRF, IP-VRF, and ARP (in the case of 
asymmetric IRB) tables to point to the new NVE. Furthermore, upon receiving the MAC/IP 
withdraw for the TS from the old NVE, these remote PEs perform the cleanups for their BGP tables. 


8. BGP Encoding 


This document defines one new BGP Extended Community for EVPN. 


8.1. EVPN Router's MAC Extended Community 


Anew EVPN BGP Extended Community called "EVPN Router's MAC" is introduced here. This new 
extended community is a transitive extended community witha Type field of 0x06 (EVPN) anda 
Sub-Type field of 0x03. It may be advertised along with the Encapsulation Extended Community 
defined in Section 4.1 of [RFC9012]. 


The EVPN Router's MAC Extended Community is encoded as an 8-octet value as follows: 


Q 1 2 3 
B12345678 90172345 6789017123456782 ead 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 

| Type=0x06 | Sub-Type=0x03 | EVPN Router's MAC 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
| EVPN Router's MAC Cont'd 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 


Figure 5: EVPN Router's MAC Extended Community 


This extended community is used to carry the PE's MAC address for symmetric IRB scenarios, and 
it is sent with EVPN RT-2. The advertising PE SHALL only attach a single EVPN Router's MAC 
Extended Community to a route. In case the receiving PE receives more than one EVPN Router's 
MAC Extended Community with a route, it SHALL process the first one in the list and not store 
and propagate the others. 


Sajassi, et al. Standards Track Page 21 


RFC 9135 IRB EVPN October 2021 


9. Operational Models for Symmetric Inter-Subnet 
Forwarding 


The following sections describe two main symmetric IRB forwarding scenarios (within a DC - i.e., 
intra-DC) along with the corresponding procedures. In the following scenarios, without loss of 
generality, it is assumed that a given tenant is represented by a single IP-VPN instance. Therefore, 
on a given PE, a tenant is represented by a single IP-VRF table and one or more MAC-VFF tables. 


9.1. IRB Forwarding on NVEs for Tenant Systems 


This section covers the symmetric IRB procedures for the scenario where each TS is attached to 
one or more NVEs, and its host IP and MAC addresses are learned by the attached NVEs and are 
distributed to all other NVEs that are interested in participating in both intra-subnet and inter- 

subnet communications with that TS. 


In this scenario, without loss of generality, it is assumed that NVEs operate in VLAN-based service 
interface mode with one bridge table(s) per MAC-VRF. Thus, for a given tenant, an NVE has one 
MAC-VRF for each tenant subnet (e.g., each VLAN) that is configured for extension via VXLAN or 
NVGRE encapsulation. In the case of VLAN-aware bundling, each MAC-VRF consists of multiple 
bridge tables (e.g., one bridge table per VLAN). The MAC-VRFs on an NVE for a given tenant are 
associated with an IP-VRF corresponding to that tenant (or IP-VPN instance) via their IRB 
interfaces. 


Since VXLAN and NVGRE encapsulations require an inner Ethernet header (inner MAC SA/DA) 
and since a TS MAC address cannot be used for inter-subnet traffic, the ingress NVE's MAC address 
is used as an inner MAC SA. The NVE's MAC address is the device MAC address, and it is common 
across all MAC-VRFs and IP-VRFs. This MAC address is advertised using the new EVPN Router's 
MAC Extended Community (Section 8.1). 


Figure 6 below illustrates this scenario, where a given tenant (e.g., an IP-VPN instance) has three 
subnets represented by MAC-VRF1, MAC-VRF2, and MAC-VRF3 across two NVEs. There are five TSs 
that are associated with these three MAC-VRFs -- i.e., TS1, TS4, and TS5 are on the same subnet (e.g., 
the same MAC-VRF/VLAN). TS1 and TS5 are associated with MAC-VRF1 on NVE1, while TS4 is 
associated with MAC-VRF1 on NVE2. TS2 is associated with MAC-VRF2 on NVE1, and TS3 is 
associated with MAC-VRF3 on NVE2. MAC-VRF1 and MAC-VRF2 on NVE1 are, in turn, associated 
with IP-VRF1 on NVE1, and MAC-VRF1 and MAC-VRF3 on NVE2 are associated with IP-VRF1 on 
NVEZ2. When TS1, TS5, and TS4 exchange traffic with each other, only the L2 forwarding (bridging) 
part of the IRB solution is exercised because all these TSs belong to the same subnet. However, 
when TS1 wants to exchange traffic with TS2 or TS3, which belong to different subnets, both the 
bridging and routing parts of the IRB solution are exercised. The following subsections describe 
the control and data plane operations for this IRB scenario in detail. 
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NVE1 mo + 
TEE + | | 
TSi- | MACx | | | NVE2 
(M1/IP1) | (MAC- | | (eee + 
TS5----- | VRF1)\ | | MPLS/ |  |MACy (MAC- |----- TS3 
(M5/IP5) | \ | VXLAN/ | | / VRF3) | (M3/IP3) 
| (IP-VRF1)|----| NVGRE |---| (IP-VRF1) | 
| if | | | | \ | 
[Specs | (MAC- / | | | | (MAC- |----- TS4 
(M2/IP2) | VRF2) | | | | VRF1) | (M4/IP4) 
+------------- + l | +------------- + 
| | 
+--------- + 


Figure 6: IRB Forwarding on NVEs for Tenant Systems 


9.1.1. Control Plane Operation 


Each NVE advertises a MAC/IP Advertisement route (i.e., route type 2) for each of its TSs with the 
following field set: 


e RD and Ethernet Segment Identifier (ESI) per [RFC7432] 

e Ethernet Tag = 0 (assuming VLAN-based service) 

e MAC Address Length = 48 

e MAC Address = Mi (where i = 1, 2, 3, 4, or 5) in Figure 6, above 
e IP Address Length = 32 or 128 

e IP Address = IPi (where i = 1, 2, 3, 4, or 5) in Figure 6, above 

e Labell = MPLS label or VNI corresponding to MAC-VRF 

e Label2 = MPLS label or VNI corresponding to IP-VRF 


Each NVE advertises an EVPN RT-2 route with two Route Targets (one corresponding to its MAC- 
VRF and the other corresponding to its IP-VRF). Furthermore, the EVPN RT-2 is advertised with 
two BGP Extended Communities. The first BGP Extended Community identifies the tunnel type, 
and it is called "Encapsulation Extended Community" as defined in [RFC9012], and the second 
BGP Extended Community includes the MAC address of the NVE (e.g., MACx for NVE1 or MACy for 
NVEZ2) as defined in Section 8.1. The EVPN Router's MAC Extended Community MUST be added 
when the Ethernet NVO tunnel is used. If the IP NVO tunnel type is used, then there is no need to 
send this second Extended Community. It should be noted that the IP NVO tunnel type is only 
applicable to symmetric IRB procedures. 


Upon receiving this advertisement, the receiving NVE performs the following: 


e It uses Route Targets corresponding to its MAC-VRF and IP-VRF for identifying these tables 
and subsequently importing the MAC and IP addresses into them, respectively. 

e It imports the MAC address from the MAC/IP Advertisement route into the MAC-VRF with the 
BGP next-hop address as the underlay tunnel destination address (e.g., VTEP DA for VXLAN 


encapsulation) and label1 as the VNI for VXLAN encapsulation or an EVPN label for MPLS 
encapsulation. 
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e If the route carries the new EVPN Router's MAC Extended Community and if the receiving 
NVE uses an Ethernet NVO tunnel, then the receiving NVE imports the IP address into IP-VRF 
with NVE's MAC address (from the new EVPN Router's MAC Extended Community) as the 
inner MAC DA, the BGP next-hop address as the underlay tunnel destination address, the 
VTEP DA for VXLAN encapsulation, and label2 as the IP-VPN VNI for VXLAN encapsulation. 


e If the receiving NVE uses MPLS encapsulation, then the receiving NVE imports the IP address 
into IP-VRF with the BGP next-hop address as the underlay tunnel destination address and 
label2 as the IP-VPN label for MPLS encapsulation. 


If the receiving NVE receives an EVPN RT-2 with only label1 and only a single Route Target 
corresponding to IP-VRF; an EVPN RT-2 with only a single Route Target corresponding to MAC- 
VRF but with both label1 and label2; or an EVPN RT-2 with a MAC address length of zero, then it 
MUST use the treat-as-withdraw approach [RFC7606] and SHOULD log an error message. 


9.1.2. Data Plane Operation 


The following description of the data plane operation describes just the logical functions, and the 
actual implementation may differ. Let's consider the data plane operation when TS1 in subnet-1 
(MAC-VRF1) on NVE1 wants to send traffic to TS3 in subnet-3 (MAC-VRF3) on NVE2. 


e NVE1 receives a packet with the MAC DA corresponding to the MAC-VRF1 IRB interface on 
NVE1 (the interface between MAC-VRF1 and IP-VRF1) and the VLAN tag corresponding to 
MAC-VRF1. 


e Upon receiving the packet, the NVE1 uses the VLAN tag to identify the MAC-VRF1. It then looks 
up the MAC DA and forwards the frame to its IRB interface. 


e The Ethernet header of the packet is stripped, and the packet is fed to the IP-VRF, where an IP 
lookup is performed on the destination IP address. NVE1 also decrements the TTL / hop limit 
for that packet by one, and if it reaches zero, NVE1 discards the packet. This lookup yields the 
outgoing NVO tunnel and the required encapsulation. If the encapsulation is for the Ethernet 
NVO tunnel, then it includes the egress NVE's MAC address as the inner MAC DA, the egress 
NVE's IP address (e.g., BGP next-hop address) as the VTEP DA, and the VPN-ID as the VNI. The 
inner MAC SA and VTEP SA are set to NVE's MAC and IP addresses, respectively. If it is an 
MPLS encapsulation, then the corresponding EVPN and LSP labels are added to the packet. 
The packet is then forwarded to the egress NVE. 


e If the egress NVE receives a packet from the Ethernet NVO tunnel (e.¢., it is VXLAN 
encapsulated), then it removes the Ethernet header. Since the inner MAC DA is the egress 
NVE's MAC address, the egress NVE knows that it needs to perform an IP lookup. It uses the 
VNI to identify the IP-VRF table. If the packet is MPLS encapsulated, then the EVPN label 
lookup identifies the IP-VRF table. Next, an IP lookup is performed for the destination TS 
(TS3), which results in an access-facing IRB interface over which the packet is sent. Before 
sending the packet over this interface, the ARP table is consulted to get the destination TS's 
MAC address. NVE2 also decrements the TTL / hop limit for that packet by one, and if it 
reaches zero, NVE2 discards the packet. 


° The IP packet is encapsulated with an Ethernet header, with the MAC SA set to that of the IRB 
interface MAC address (i.e., the IRB interface between MAC-VRF3 and IP-VRF1 on NVEZ2) and 
the MAC DA set to that of the destination TS (TS3) MAC address. The packet is sent to the 
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corresponding MAC-VRF (i.e., MAC-VRF3) and, after a lookup of MAC DA, is forwarded to the 
destination TS (TS3) over the corresponding interface. 


In this symmetric IRB scenario, inter-subnet traffic between NVEs will always use the IP-VRF VNI/ 
MPLS label. For instance, traffic from TS2 to TS4 will be encapsulated by NVE1 using NVE2's IP- 
VRF VNI/MPLS label, as long as TS4's host IP is present in NVE1's IP-VRF. 


9.2. IRB Forwarding on NVEs for Subnets behind Tenant Systems 


This section covers the symmetric IRB procedures for the scenario where some TSs support one or 
more subnets and these TSs are associated with one or more NVEs. Therefore, besides the 
advertisement of MAC/IP addresses for each TS, which can be multihomed with All-Active 
redundancy mode, the associated NVE needs to also advertise the subnets statically configured 
on each TS. 


The main difference between this solution and the previous one is the additional advertisement 
corresponding to each subnet. These subnet advertisements are accomplished using the EVPN IP 
Prefix route defined in [RFC9136]. These subnet prefixes are advertised with the IP address of their 
associated TS (which is in an overlay address space) as their next hop. The receiving NVEs 
perform recursive route resolution to resolve the subnet prefix with its advertising NVE so that 
they know which NVE to forward the packets to when they are destined for that subnet prefix. 


The advantage of this recursive route resolution is that when a TS moves from one NVE to 
another, there is no need to re-advertise any of the subnet prefixes for that TS. All that is needed is 
to advertise the IP/MAC addresses associated with the TS itself and exercise the MAC Mobility 
procedures for that TS. The recursive route resolution automatically takes care of the updates for 
the subnet prefixes of that TS. 


Figure 7 illustrates this scenario where a given tenant (e.g.,an IP-VPN service) has three subnets 
represented by MAC-VRF1, MAC-VRF2, and MAC-VRF3 across two NVEs. There are four TSs 
associated with these three MAC-VRFs -- i.e., TS1 is connected to MAC-VRF1 on NVE1, TS2 is 
connected to MAC-VRF2 on NVE1, TS3 is connected to MAC-VRF3 on NVE2, and TS4 is connected 
to MAC-VRF1 on NVE2. TS1 has two subnet prefixes (SN1 and SN2), and TS3 has a single subnet 
prefix (SN3). The MAC-VRFs on each NVE are associated with their corresponding IP-VRF using 
their IRB interfaces. When TS4 and TS1 exchange intra-subnet traffic, only the L2 forwarding 
(bridging) part of the IRB solution is used (i.e., the traffic only goes through their MAC-VRFs); 
however, when TS3 wants to forward traffic to SN1 or SN2 sitting behind TS1 (inter-subnet traffic), 
then both the bridging and routing parts of the IRB solution are exercised (i.e., the traffic goes 
through the corresponding MAC-VRFs and IP-VRFs). If TS4, for example, wants to reach SN1, it 
uses its default route and sends the packet to the MAC address associated with the IRB interface 
on NVE2; NVE2 then performs an IP lookup in its IP-VRF and finds an entry for SN1. The following 
subsections describe the control and data plane operations for this IRB scenario in detail. 
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NVE1 +---------- + 
SN1--+ +------------- + | 
|--TS1----- | (MAC- \ | | 
SN2--+ M1/IP1 | VRF1) \ | | 
| CUP SVR) lisa | 
| / | l | 
TS2----- | (MAC- / | | MPLS/ | 
M2/IP2 | VRF2) | | VXLAN/ | 
+------------- + | NVGRE | 
eee + | | 
SN3--+--TS3----- | (MAC-\ | | 
M3 /1IP3 | VRF3)\ | | | 
| (IP=VRE) [=== | 
| / | l | 
TS4----- | (MAC- / | | | 
M4/IP4 | VRF1) | | | 
+------------- + +---------- + 
NVE2 


Figure 7: IRB Forwarding on NVEs for Subnets behind TSs 


Note that in Figure 7, above, SN1 and SN2 are configured on NVE1, which then advertises each in 
an IP Prefix route. Similarly, SN3 is configured on NVE2, which then advertises it in an IP Prefix 
route. 


9.2.1. Control Plane Operation 


Each NVE advertises a route type 5 (EVPN RT-5, IP Prefix route defined in [RFC9136]) for each of its 
subnet prefixes with the IP address of its TS as the next hop (Gateway Address field) as follows: 


e RD associated with the IP-VRF 

e ESI =0 

e Ethernet Tag =0 

e IP Prefix Length = 0 to 32 or 0 to 128 

IP Prefix = SNi 

e Gateway Address = IPi (IP address of TS) 
e MPLS Label = 0 


This EVPN RT-5 is advertised with one or more Route Targets associated with the IP-VRF from 
which the route is originated. 


Each NVE also advertises an EVPN RT-2 (MAC/IP Advertisement route) along with its associated 
Route Targets and Extended Communities for each of its TSs exactly as described in Section 9.1.1. 


Upon receiving the EVPN RT-5 advertisement, the receiving NVE performs the following: 


e It uses the Route Target to identify the corresponding IP-VRF. 


e It imports the IP prefix into its corresponding IP-VRF configured with an import RT that is one 
of the RTs being carried by the EVPN RT-5 route, along with the IP address of the associated TS 
as its next hop. 
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When receiving the EVPN RT-2 advertisement, the receiving NVE imports the MAC/IP addresses of 
the TS into the corresponding MAC-VRF and IP-VRF per Section 9.1.1. When both routes exist, 
recursive route resolution is performed to resolve the IP prefix (received in EVPN RT-5) to its 
corresponding NVE's IP address (e.g., its BGP next hop). The BGP next hop will be used as the 
underlay tunnel destination address (e.g., VTEP DA for VXLAN encapsulation), and the EVPN 
Router's MAC will be used as the inner MAC for VXLAN encapsulation. 


9.2.2. Data Plane Operation 


The following description of the data plane operation describes just the logical functions, and the 
actual implementation may differ. Let's consider the data plane operation when a host in SN1 
behind TS1 wants to send traffic to a host in SN3 behind TS3. 


e TS1 sends a packet with MAC DA corresponding to the MAC-VRF1 IRB interface of NVE1 anda 
VLAN tag corresponding to MAC-VRF1. 


e Upon receiving the packet, the ingress NVE1 uses the VLAN tag to identify the MAC-VRF1. It 
then looks up the MAC DA and forwards the frame to its IRB interface as in Section 9.1.1. 


e The Ethernet header of the packet is stripped, and the packet is fed to the IP-VRF, where an IP 
lookup is performed on the destination address. This lookup yields the fields needed for 
VXLAN encapsulation with NVE2's MAC address as the inner MAC DA, NVE2's IP address as the 
VTEP DA, and the VNI. The MAC SA is set to NVE1's MAC address, and the VTEP SA is set to 
NVE1's IP address. NVE1 also decrements the TTL/ hop limit for that packet by one, and if it 
reaches zero, NVE1 discards the packet. 


e The packet is then encapsulated with the proper header based on the above info and is 
forwarded to the egress NVE (NVE2). 


e On the egress NVE (NVE2), assuming the packet is VXLAN encapsulated, the VXLAN and the 
inner Ethernet headers are removed, and the resultant IP packet is fed to the IP-VRF 
associated with that VNI. 

e Next, a lookup is performed based on the IP DA (which is in SN3) in the associated IP-VRF of 
NVE2. The IP lookup yields the access-facing IRB interface over which the packet needs to be 
sent. Before sending the packet over this interface, the ARP table is consulted to get the 
destination TS (TS3) MAC address. NVE2 also decrements the TTL / hop limit for that packet by 
one, and if it reaches zero, NVE2 discards the packet. 

e The IP packet is encapsulated with an Ethernet header with the MAC SA set to that of the 
access-facing IRB interface of the egress NVE (NVE2), and the MAC DA is set to that of the 
destination TS (TS3) MAC address. The packet is sent to the corresponding MAC-VRF3 and, 
after a lookup of MAC DA, is forwarded to the destination TS (TS3) over the corresponding 
interface. 


10. Security Considerations 


The security considerations for Layer 2 forwarding in this document follow those of [RFC7432] for 
MPLS encapsulation and those of [RFC8365] for VXLAN or NVGRE encapsulations. This section 
describes additional considerations. 
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This document describes a set of procedures for inter-subnet forwarding of tenant traffic across 
PEs (or NVEs). These procedures include both Layer 2 forwarding and Layer 3 routing on a packet- 
by-packet basis. The security consideration for Layer 3 routing in this document follows that of 
[RFC4365], with the exception of the application of routing protocols between CEs and PEs. 
Contrary to [RFC4364], this document does not describe route distribution techniques between 
CEs and PEs but rather considers the CEs as TSs or VAs that do not run dynamic routing protocols. 
This can be considered a security advantage, since dynamic routing protocols can be blocked on 
the NVE/PE ACs, not allowing the tenant to interact with the infrastructure's dynamic routing 
protocols. 


The VPN scheme described in this document does not provide the quartet of security properties 
mentioned in [RFC4365] (confidentiality protection, source authentication, integrity protection, 
and replay protection). If these are desired, they must be provided by mechanisms that are 
outside the scope of the VPN mechanisms. 


In this document, the EVPN RT-5 is used for certain scenarios. This route uses an Overlay Index 
that requires a recursive resolution to a different EVPN route (an EVPN RT-2). Because of this, it is 
worth noting that any action that ends up filtering or modifying the EVPN RT-2 route used to 
convey the Overlay Indexes will modify the resolution of the EVPN RT-5 and therefore the 
forwarding of packets to the remote subnet. 


11. IANA Considerations 


IANA has allocated Sub-Type value 0x03 in the "EVPN Extended Community Sub-Types" registry as 
follows: 


Sub-Type Value Name Reference 
0x03 EVPN Router's MAC Extended Community RFC 9135 
Table 1 


This document has been listed as an additional reference for the MAC/IP Advertisement route in 
the "EVPN Route Types" registry. 
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